Flexible illumination for imaging systems

ABSTRACT

Methods and apparatus for flexible illumination that improve the performance and robustness of an imaging system are described. Multiple different lighting configurations for the imaging system are pre-generated. Each lighting configuration may specify one or more aspects of lighting. A lookup table may be generated via which each pose is associated with a respective lighting configuration. A user may put on, hold, or otherwise use the device. A process may be initiated in which different lighting configurations may be selected by the controller to capture images of the user&#39;s eye, periorbital region, or face at different poses and in different conditions for use by a biometric authentication or gaze tracking process.

This application is a 371 of PCT Application No. PCT/US2021/051611,filed Sep. 22, 2021, which claims benefit of priority to U.S.Provisional Patent Application No. 63/083,718, filed Sep. 25, 2020. Theabove applications are incorporated herein by reference. To the extentthat any material in the incorporated application conflicts withmaterial expressly set forth herein, the material expressly set forthherein controls.

An eye or gaze tracker is a device for estimating eye positions and eyemovement. Eye tracking systems have been used in research on the visualsystem, in psychology, psycholinguistics, marketing, and as inputdevices for human-computer interaction. In the latter application,typically the intersection of a person's point of gaze with a desktopmonitor is considered.

Biometric authentication technology uses one or more features of aperson to identify that person, for example for secure, authenticatedaccess to devices, systems, or rooms. In a typical scenario, in aregistration process one or more images are captured of the featuresbeing tracked (e.g., images of a person's iris(es)), and the images areprocessed to generate a set or vector of metrics that are unique to, andthus uniquely identify, that person. When the person attempts to accessthe device, system, or room, images of the person's features are againcaptured and processed using a similar algorithm to the one used duringregistration. The extracted metrics are compared to the baseline metricsand, if the match is sufficiently good, the person is allowed access.

SUMMARY

Embodiments of imaging systems that implement flexible illuminationmethods are described. Embodiments may provide methods that improve theperformance and robustness of an imaging system, and that make theimaging system adaptable to specific users, conditions, and setup forbiometric authentication using the eyes and periorbital region, gazetracking, and anti-spoofing. While, conventional eye tracking systemsfocus on specular reflections or glints for gaze tracking, embodimentsmay focus on other aspects such as providing uniform, good contrast onthe iris or other regions of interest, reducing or illuminating shadowson regions of interest, and other improvements for biometricauthentication applications.

In embodiments, two or more different lighting configurations for theimaging system in a device are pre-generated. Each lightingconfiguration may specify one or more aspects of lighting including, butnot limited to, which LEDs or group of LEDs to enable or disable,intensity/brightness, wavelength, shapes and sizes of the lights,direction, sequences of lights, etc. One or more lighting configurationsmay be generated for each of two or more poses, where a pose is a 3Dgeometrical relationship between the eye camera and the user's currenteye position and gaze direction. A lookup table may be generated viawhich each pose is associated with its respective lightingconfiguration(s). The lookup table and lighting configurations may, forexample be stored to memory of the device and/or to memory accessible tothe device via a wired or wireless connection

In some embodiments, the lighting configurations may be pre-generatedsynthetically for a device and imaging system, for example using a 3Dgeometric model or representation of the device and imaging system togenerate lighting configurations for a set of estimated poses.Alternatively, in some embodiments, the lighting configurations may bepre-generated using a data set of images of real-world user faces toobtain pose information. As another alternative, in some embodiments,the lighting configurations may be generated during an initializationprocess for a particular user. For example, in some embodiments, theuser puts on or holds the device and moves their gaze around, and thesystem/controller runs through a process during which images arecaptured and processed with different light settings to determineoptimal lighting configurations for this user when capturing images ofthe desired features at two or more different poses.

In some embodiments, after the lighting configurations and lookup tableare generated, the user may put on, hold, or otherwise use the device. Abiometric authentication process may be initiated in which differentlighting configurations may be selected by the controller to captureoptimal images of the desired features of the user's eye (e.g., iris,periorbital region, etc.) at different poses and in different conditionsfor use by the biometric authentication algorithms executed by thecontroller.

In some embodiments, the device may initiate a biometric authenticationprocess when the user accesses the device. In some embodiments, thedevice's controller may begin the biometric authentication process witha default initial lighting configuration. One or more images may becaptured by the imaging system using the respective setting for theillumination source, and the captured image(s) may be checked forquality. If the images are satisfactory for the algorithms that processthe images to perform biometric authentication using one or morefeatures of the user's eye, periorbital region, and/or other facialfeatures, then the flexible illumination process may be done. Otherwise,the controller may select another lighting configuration, direct theillumination source to illuminate the subject according to the newlighting configuration, and direct the camera to capture one or moreimages that are checked for quality. This process may be repeated untila successful authentication has been achieved, or for a specified numberof attempts until the authentication attempt is considered failed. Insome embodiments, the user's current pose may be determined by theimaging system and controller, for example using a gaze trackingalgorithm, and the user's current pose may be used to select an initiallighting configuration and, if necessary, one or more subsequentlighting configurations for the biometric authentication process.

A similar method may be applied in a gaze tracking process in whichdifferent lighting configurations are selected by the controller toobtain better images of the desired features of the user's eyes (e.g.,glints) at different poses and in different conditions.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A through 1D illustrate example eye camera systems, according tosome embodiments.

FIG. 2 graphically illustrates tradeoffs between complexities in abiometric authentication system, according to some embodiments.

FIG. 3 is a block diagram of an imaging systems that implements aflexible illumination method, according to some embodiments.

FIG. 4 is a flowchart of a method for providing flexible illumination inan imaging system, according to some embodiments.

FIGS. 5A and 5B illustrate a biometric authentication system thatcombines different biometric aspects, according to some embodiments.

FIG. 6 is a flowchart of a method for performing biometricauthentication using multiple biometric aspects, according to someembodiments.

FIG. 7 illustrates a biometric authentication system that uses multiplecameras, according to some embodiments.

FIG. 8A is a flowchart of a method for biometric authentication usingmultiple cameras, according to some embodiments.

FIG. 8B is a flowchart of another method for biometric authenticationusing multiple cameras, according to some embodiments.

FIG. 9A illustrates a system that includes at least one additionaloptical element on the light path between the user's eye and the eyecamera, according to some embodiments.

FIG. 9B illustrates a system that includes a diffractive optical elementon the light path between the user's eye and the eye camera to improvethe viewing angle of the camera, according to some embodiments.

FIG. 10 is a flowchart of a method for processing images in a systemthat includes at least one additional optical element on the light pathbetween the user's eye and the eye camera, according to someembodiments.

FIG. 11 is a flowchart of a method for capturing and processing imagesin a system that includes a diffractive optical element on the lightpath between the user's eye and the eye camera to improve the viewingangle of the camera, according to some embodiments.

FIGS. 12A through 12C illustrate a system that includes light sourcesthat emit light at multiple wavelengths to sequentially capture imagesat the multiple wavelengths, according to some embodiments.

FIGS. 13A and 13B illustrate a system that includes a camera with aphotosensor that concurrently captures multiple images at differentwavelengths, according to some embodiments.

FIG. 14 is a flowchart of a method for sequentially capturing andprocessing images at multiple wavelengths, according to someembodiments.

FIG. 15 is a flowchart of a method for concurrently capturing andprocessing images at multiple wavelengths, according to someembodiments.

FIG. 16 illustrates a system that provides feedback to the user and/orcontrol signals to the imaging system to manually or mechanically adjustthe viewing angle of the camera with respect to the user's eye orperiocular region, according to some embodiments.

FIG. 17 is a flowchart of a method for providing feedback to the user tomanually adjust the viewing angle of the camera with respect to theuser's eye or periocular region, according to some embodiments.

FIG. 18 is a flowchart of a method for providing control signals to theimaging system to mechanically adjust the viewing angle of the camerawith respect to the user's eye or periocular region, according to someembodiments.

FIGS. 19A and 19B are block diagrams illustrating a device that mayinclude components and implement methods as illustrated in FIGS. 1through 18 , according to some embodiments.

FIG. 20 illustrates an example head-mounted device (HMD) that mayinclude components and implement methods as illustrated in FIGS. 1through 18 , according to some embodiments.

FIG. 21 is a block diagram illustrating an example system that mayinclude components and implement methods as illustrated in FIGS. 1through 18 , according to some embodiments.

This specification includes references to “one embodiment” or “anembodiment.” The appearances of the phrases “in one embodiment” or “inan embodiment” do not necessarily refer to the same embodiment.Particular features, structures, or characteristics may be combined inany suitable manner consistent with this disclosure.

“Comprising.” This term is open-ended. As used in the claims, this termdoes not foreclose additional structure or steps. Consider a claim thatrecites: “An apparatus comprising one or more processor units . . . . ”Such a claim does not foreclose the apparatus from including additionalcomponents (e.g., a network interface unit, graphics circuitry, etc.).

“Configured To.” Various units, circuits, or other components may bedescribed or claimed as “configured to” perform a task or tasks. In suchcontexts, “configured to” is used to connote structure by indicatingthat the units/circuits/components include structure (e.g., circuitry)that performs those task or tasks during operation. As such, theunit/circuit/component can be said to be configured to perform the taskeven when the specified unit/circuit/component is not currentlyoperational (e.g., is not on). The units/circuits/components used withthe “configured to” language include hardware—for example, circuits,memory storing program instructions executable to implement theoperation, etc. Reciting that a unit/circuit/component is “configuredto” perform one or more tasks is expressly intended not to invoke 35U.S.C. § 112, paragraph (f), for that unit/circuit/component.Additionally, “configured to” can include generic structure (e.g.,generic circuitry) that is manipulated by software or firmware (e.g., anFPGA or a general-purpose processor executing software) to operate inmanner that is capable of performing the task(s) at issue. “Configureto” may also include adapting a manufacturing process (e.g., asemiconductor fabrication facility) to fabricate devices (e.g.,integrated circuits) that are adapted to implement or perform one ormore tasks.

“First,” “Second,” etc. As used herein, these terms are used as labelsfor nouns that they precede, and do not imply any type of ordering(e.g., spatial, temporal, logical, etc.). For example, a buffer circuitmay be described herein as performing write operations for “first” and“second” values. The terms “first” and “second” do not necessarily implythat the first value must be written before the second value.

“Based On” or “Dependent On.” As used herein, these terms are used todescribe one or more factors that affect a determination. These terms donot foreclose additional factors that may affect a determination. Thatis, a determination may be solely based on those factors or based, atleast in part, on those factors. Consider the phrase “determine A basedon B.” While in this case, B is a factor that affects the determinationof A, such a phrase does not foreclose the determination of A from alsobeing based on C. In other instances, A may be determined based solelyon B.

“Or.” When used in the claims, the term “or” is used as an inclusive orand not as an exclusive or. For example, the phrase “at least one of x,y, or z” means any one of x, y, and z, as well as any combinationthereof.

DETAILED DESCRIPTION

Various embodiments of methods and apparatus for flexible illuminationin imaging systems are described. An imaging system as described hereinmay include two or more illumination sources (e.g., point light sourcessuch as light-emitting diodes (LEDs)) that illuminate an object to beimaged (e.g., a person's eye or eye region), and at least one cameraconfigured to capture images of light from the illumination sourcesreflected by the object when illuminated.

Embodiments of the imaging system may, for example, be used forbiometric authentication, for example using features of the user's eyessuch as the iris, the eye region (referred to as the periocular region),or other parts of the user's face such as the eyebrows. A biometricauthentication system uses one or more of the features to identify aperson, for example for secure, authenticated access to devices,systems, or rooms. In a registration process one or more images arecaptured of the features being tracked (e.g., images of a person'siris(es), periocular region, etc.), and the images are processed togenerate a set or vector of metrics that are unique to, and thusuniquely identify, that person. When the person attempts to access thedevice, system, or room, images of the person's features are againcaptured and processed using a similar algorithm to the one used duringregistration. The extracted metrics are compared to the baseline metricsand, if the match is sufficiently good, the person may be allowedaccess.

Another example use for embodiments of the imaging system is in gazetracking. A gaze tracking system may, for example, be used to computegaze direction and a visual axis using glints and eye features based ona three-dimensional (3D) geometric model of the eye.

Embodiments of the imaging system described herein may, for example, beused in a biometric authentic process, a gaze tracking process, or both.Another example is in anti-spoofing, which is related to biometricauthentication in that “spoofing” refers to attempts to trick abiometric authentication system by, for example, presenting a picture ormodel of a valid user's eye, eye region, or face. More generally,embodiments of the imaging system may be implemented in any applicationor system in which images of an object illuminated by a light source arecaptured by one or more cameras for processing.

A non-limiting example application of the methods and apparatus forflexible illumination in imaging systems are in systems that include atleast one eye camera (e.g., infrared (IR) cameras) positioned at eachside of a user's face, and an illumination source (e.g., point lightsources such as an array or ring of IR light-emitting diodes (LEDs))that emit light towards the user's eyes. The imaging system may, forexample, be a component of a head-mounted device (HMD), for example aHMD of an extended reality (XR) system such as a mixed or augmentedreality (MR) system or virtual reality (VR) system. The HMD may, forexample be implemented as a pair of glasses, googles, or helmet. Otherexample applications for the imaging system include mobile devices suchas smartphones, pad or tablet devices, desktop computers, and notebookcomputers, as well as stand-alone biometric authentication systemsmounted on walls or otherwise located in rooms or on buildings. In anyof these example systems, the imaging system may be used for biometricauthentication, gaze tracking, or both.

FIGS. 1A through 1D illustrate example imaging systems, according tosome embodiments. The imaging system may include, but is not limited to,one or more cameras 140, an illumination source 130, and a controller160. FIG. 1A shows an imaging system in in which the eye camera 140images the eye 192 directly. However, in some embodiments the eye camera140 may instead image a reflection of the eye 192 off of a hot mirror150 as shown in FIG. 1B. In addition, in some embodiments, the eyecamera 140 may image the eye through a lens 120 of an imaging system,for example as shown in FIG. 1C.

In some embodiments, a device (e.g., a head-mounted device (HMD)) mayinclude an imaging system that includes at least one eye camera 140(e.g., infrared (IR) cameras) positioned on one side or at each side ofthe user's face, and an illumination source 130 (e.g., point lightsources such as an array or ring of IR light-emitting diodes (LEDs))that emits light towards the user's eye(s) 192 or periorbital region.

FIG. 1D shows an example illumination source 130 that includes multipleLEDs 132. In this example, there are eight LEDs 132 arranged in a ring.Note, however, that the number and arrangement of the LEDS 132 in anillumination source 130 may be different. In addition, in someembodiments other light-emitting elements than LEDs may be used. In someembodiments, the LEDs 132 may be configured to emit light in the IRrange, including SWIR and/or NIR, for example at 740, 750, 840, 850,940, or 950 nanometers.

The eye camera 140 may be pointed towards the eye 192 to receive lightfrom the illumination source 130 reflected from the eye 192, as shown inFIG. 1A. However, in some embodiments the eye camera 140 may insteadimage a reflection of the eye 192 off of a hot mirror 150 as shown inFIG. 1B. In addition, in some embodiments, the eye camera 140 may imagethe eye 192 through a lens 120 or other optical element of the device,for example as shown in FIG. 1C.

The device that includes the imaging system may include a controller 160comprising one or more processors and memory. Controller 160 may includeone or more of various types of processors, image signal processors(ISPs), graphics processing units (GPUs), coder/decoders (codecs),and/or other components for processing and rendering video and/orimages. In some embodiments, the controller 160 may be integrated in thedevice. In some embodiments, at least some of the functionality of thecontroller 160 may be implemented by an external device coupled to thedevice by a wired or wireless connection. While not shown in FIGS. 1Athrough 1C, in some embodiments controller 160 may be coupled to anexternal memory for storing and reading data and/or software.

The controller 160 may send control signals to the illumination source130 and camera 140 to control the illumination of the eye 192 andcapture of images of the eye 192. The controller 160 may use input 142(e.g., captured images of the eyes 192) from the eye cameras 140 forvarious purposes, for example for biometric authentication or gazetracking. The controller 160 may implement algorithms that estimate theuser's gaze direction based on the input 142. For example, thecontroller 160 may implement algorithms that process images captured bythe cameras 140 to identify features of the eye 192 (e.g., the pupil,iris, and sclera) or periorbital region to be used in biometricauthentication algorithms. As another example, the controller 160 mayimplement gaze tracking algorithms that process images captured by thecameras 140 to identify glints (reflections of the LEDs 130) obtainedfrom the eye cameras 140. The information obtained from the input 142may, for example, be used to determine the direction in which the useris currently looking (the gaze direction), and may be used to constructor adjust a 3D model of the eye 192.

However, in a device that implements the imaging system, components ofthe device may result in unwanted reflections and stray light on thefinal image captured by camera 140. As the imaging system becomes morecomplex, for example with optical surfaces (e.g., lenses 120 and/ormirrors 150) involved in the trajectory between the point light sources130 and camera 140, the higher the likelihood of getting unwantedreflections and stray light on the final image captured by camera 140,for example caused by reflections in lenses, imperfections in lenses oroptical surfaces, or dust on optical surfaces. When using the imagingfor biometric authentication and/or for gaze tracking, components of thedevice (e.g., lenses) may block, refract, or reflect light, including aportion of the light from the illumination source 130 and ambient light,if present. In addition, position of the device and imaging system withrespect to the user's head may shift during use. Other aspects of thedevice and imaging system may change. For example, a surface of a lensin the device may become smudged, or the user may add or changesomething such as clip-on lenses to the device. Thus, quality of theimages captured with the imaging system may vary depending on thecurrent lighting conditions, position of the device and imaging systemwith respect to the user's head, and other factors such as smudges orother changes to the device. The quality of the captured images mayaffect the efficiency and accuracy of algorithms used in variousapplications including but not limited to biometric authentication,anti-spoofing, and gaze tracking.

Embodiments of the methods and apparatus for flexible illumination inimaging systems as described herein may improve the performance androbustness of an imaging system, and may help to adapt the imagingsystem to specific users, conditions, and setup for applicationsincluding but not limited to biometric authentication, anti-spoofing,and gaze tracking.

FIG. 2 graphically illustrates tradeoffs between complexities in abiometric authentication system, according to some embodiments.Embodiments of an imaging system used for biometric authentication asdescribed herein may trade off system complexity 210 for complexity inthe enrollment 200 process. A more complex system 210 may reduce thecomplexity of the enrollment process for the user, for example byautomating processes such as shifting the camera to get a better view ofthe eye rather than having the user move the device manually.Conversely, the enrollment 200 process could be made more complex toreduce system complexity 210. Similarly, biometric authentication may beimproved by increasing the number of aspects 220 of the user's eyes andperiorbital region that are used in the identification process at theexpense of system complexity 210 and possibly enrollment complexity 200.Similar tradeoffs may apply in other applications such as gaze tracking.

Flexible Illumination for Imaging Systems

Embodiments of imaging systems that implement flexible illuminationmethods are described. Embodiments may provide methods that improve theperformance and robustness of an imaging system, and that make theimaging system adaptable to specific users, conditions, and setup forbiometric authentication using the eyes and periorbital region, gazetracking, and anti-spoofing. While, conventional eye tracking systemsfocus on specular reflections or glints for gaze tracking, embodimentsmay focus on other aspects such as providing uniform, good contrast onthe iris or other regions of interest, reducing or illuminating shadowson regions of interest, and other improvements for biometricauthentication applications.

In embodiments, two or more different lighting configurations for theimaging system in a device are pre-generated. Each lightingconfiguration may specify one or more aspects of lighting including, butnot limited to, which LEDs or group of LEDs to enable or disable,intensity/brightness, wavelength, shapes and sizes of the lights,direction, sequences of lights, etc. One or more lighting configurationsmay be generated for each of two or more poses, where a pose is a 3Dgeometrical relationship between the eye camera and the user's currenteye position and gaze direction. A lookup table may be generated viawhich each pose is associated with its respective lightingconfiguration(s). The lookup table and lighting configurations may, forexample be stored to memory of the device and/or to memory accessible tothe device via a wired or wireless connection

In some embodiments, the lighting configurations may be pre-generatedsynthetically for a device and imaging system, for example using a 3Dgeometric model or representation of the device and imaging system togenerate lighting configurations for a set of estimated poses.Alternatively, in some embodiments, the lighting configurations may bepre-generated using a data set of images of real-world user faces toobtain pose information. As another alternative, in some embodiments,the lighting configurations may be generated during an initializationprocess for a particular user. For example, in some embodiments, theuser puts on or holds the device and moves their gaze around, and thesystem/controller runs through a process during which images arecaptured and processed with different light settings to determineoptimal lighting configurations for this user when capturing images ofthe desired features at two or more different poses.

In some embodiments, after the lighting configurations and lookup tableare generated, the user may put on, hold, or otherwise use the device. Abiometric authentication process may be initiated in which differentlighting configurations may be selected by the controller to captureoptimal images of the desired features of the user's eye (e.g., iris,periorbital region, etc.) at different poses and in different conditionsfor use by the biometric authentication algorithms executed by thecontroller.

In some embodiments, the device may initiate a biometric authenticationprocess when the user accesses the device. In some embodiments, thedevice's controller may begin the biometric authentication process witha default initial lighting configuration. One or more images may becaptured by the imaging system using the respective setting for theillumination source, and the captured image(s) may be checked forquality. If the images are satisfactory for the algorithms that processthe images to perform biometric authentication using one or morefeatures of the user's eye, periorbital region, and/or other facialfeatures, then the flexible illumination process may be done. Otherwise,the controller may select another lighting configuration, direct theillumination source to illuminate the subject according to the newlighting configuration, and direct the camera to capture one or moreimages that are checked for quality. This process may be repeated untila successful authentication has been achieved, or for a specified numberof attempts until the authentication attempt is considered failed. Insome embodiments, the user's current pose may be determined by theimaging system and controller, for example using a gaze trackingalgorithm, and the user's current pose may be used to select an initiallighting configuration and, if necessary, one or more subsequentlighting configurations for the biometric authentication process.

A similar method may be applied in a gaze tracking process in whichdifferent lighting configurations are selected by the controller toobtain better images of the desired features of the user's eyes (e.g.,glints) at different poses and in different conditions.

Embodiments of the flexible illumination method may improve theperformance and robustness of an imaging system, and may help to adaptthe imaging system to specific users, conditions, and setup forapplications including but not limited to biometric authentication,anti-spoofing, and gaze tracking. Embodiments may capture and processimages of the eye or periorbital region using one or more differentlighting configurations until a lighting configuration is found thatprovides optimal (or at least good enough) images to perform aparticular function (e.g., biometric authentication, gaze tracking,etc.), thus improving the performance and robustness of the device,system, and/or algorithm that uses image(s) of the eye or periorbitalregion in performing the function (e.g., biometric authentication, gazetracking, etc.).

By dynamically searching for and finding a good or optimal lightingconfiguration for current conditions, embodiments of the flexibleillumination method may help to make an imaging system adaptable to oneor more of, but not limited to:

-   -   the anatomy and appearance of users of a device or system that        includes the imaging system;    -   environmental/ambient lighting conditions;    -   reflections, streaks, ghosts, stray light, etc. that may be        visible in captured images of the eye or periorbital region;    -   changes in the optical path between at least one of the LEDs in        the illumination source, the eye or periorbital region, and at        least one eye camera, including but not limited indirect optical        paths with enclosures or other elements of a device that        includes the imaging system, which may result in additional        reflections or visual impairment of the captured images;    -   other changes in the device that includes the imaging system,        such as the addition of clip-on lenses to the device; and    -   variations in prescriptions specific to particular users that        may be used in optical elements of the device that or one the        optical path between the LEDs of the illumination source and the        eye camera(s).

Embodiments of the flexible illumination method may, for example, beimplemented in any of the illumination systems as illustrated in FIGS.1A through 1D. FIGS. 19A through 21 illustrate example devices andsystems that may include imaging systems that implement embodiments ofthe flexible illumination method. An illumination system that implementsthe flexible illumination may include, but is not limited to:

-   -   at least one eye camera (e.g., an infrared (IR) or near-infrared        (NIR) camera, an RGB or RGB-D camera, etc.); and    -   an illumination source that includes multiple light-emitting        elements which can be controlled individually or in groups        (e.g., IR or NIR LEDs, or LEDs in other wavelengths).

In embodiments, a controller of the device that includes the imagingsystem may control one or more of, but not limited to, the followingbased on a current lighting configuration:

-   -   turning individual, or groups of, the light-emitting elements on        or off;    -   increasing or decreasing the intensity/current to individual, or        groups of, the light-emitting elements; and    -   sequencing of individual, or groups of, the light-emitting        elements.

In embodiments, the light-emitting elements, or groups of thelight-emitting elements, may differ in one or more of, but not limitedto, the following:

-   -   wavelength;    -   location and orientation (pose);    -   shape;    -   size; and    -   light emitting angular profile.

In some embodiments, individual light-emitting elements or groups oflight-emitting elements may include additional optical elements, forexample lenses, grids, etc., that affect light emitted by the elementsor groups of light-emitting elements.

The following broadly describes a method for selecting a lightingconfiguration, according to some embodiments. One or more images of auser's eye or periorbital region may be captured using a first lightingconfiguration. Additional images may be captured using at least oneadditional lighting configuration. One or more objective criteria (e.g.,contrast, shadows, edges, undesirable streaks, etc.) may be selected ordetermined for analyzing the images. Based on an analysis of thecaptured images using the objective criteria, one of the lightingconfigurations that corresponds to one or more image(s) that bestsatisfies the objective criteria for this user may be selected. In someembodiments, if a change in the conditions under which the lightingconfiguration was selected is detected (e.g., some change in the user'sposition or appearance, a change in ambient lighting, a change to thedevice that includes the imaging system, etc.), then the method forselecting a lighting configuration may be repeated.

The objective criteria used in selecting lighting configurations maydiffer based on the particular application. For example, in a biometricauthentication process that uses the iris to authenticate users, thealgorithm may need images of the iris with uniform, good contrast, noshadows, etc. In a gaze tracking process, the algorithm may need imagesthat include specular reflections or glints in certain locations and/orof certain sizes and number.

In some embodiments, the objective criteria used in selecting lightingconfigurations may differ based on the environment (e.g., internal vsexternal ambient conditions). In some embodiments, the objectivecriteria used in selecting lighting configurations may differ based onvarying gaze poses or adjustments to a user's face, for example eyerelief (depth) and interpupillary distance (IPD).

FIG. 3 is a block diagram of an imaging systems that implements aflexible illumination method, according to some embodiments. Two or morelighting configurations 372 may be generated in a configurationgeneration 310 process. In some embodiments, the lighting configurationsmay be pre-generated synthetically for a device and imaging system, forexample using a 3D geometric model or representation of the device andimaging system to generate lighting configurations for a set ofestimated poses. Alternatively, in some embodiments, the lightingconfigurations may be pre-generated using a data set of images ofreal-world user faces to obtain pose information. As anotheralternative, in some embodiments, the lighting configurations may begenerated during an initialization process for a particular user. Forexample, in some embodiments, the user puts on or holds the device andmoves their gaze around, and the system/controller runs through aprocess during which images are captured and processed with differentlight settings to determine optimal lighting configurations for thisuser when capturing images of the desired features at two or moredifferent poses.

The pre-generated lighting configurations 372 may be stored 320 tomemory 370 accessible to controller 360. In some embodiments, a lookuptable 374 may be generated and stored to memory 370 that, for example,maps particular poses to particular lighting configurations.

In some embodiments, after the lighting configurations 372 and lookuptable 374 are generated and stored, a user may put on, hold, orotherwise use a device that includes the controller 360, illuminationsource 330, and eye camera(s) 340. A biometric authentication processmay be initiated in which different lighting configurations 372 may beselected by the controller 360 to capture optimal images of the desiredfeatures of the user's eye (e.g., iris, periorbital region, etc.) atdifferent poses and in different conditions for use by the biometricauthentication algorithms executed by the controller 360.

In some embodiments, the device may initiate a biometric authenticationprocess when the user accesses the device. In some embodiments, thedevice's controller 360 may begin a biometric authentication process bydirecting 344 the illumination source 330 to use a default initiallighting configuration 372. One or more images may be captured 342 bythe eye camera(s) 340 using the respective lighting provided by theillumination source 330, and the captured image(s) may be checked forquality according to one or more objective criteria or measures aspreviously described. If the images are satisfactory for the biometricauthentication algorithms that rely on one or more features of theuser's eye, periorbital region, and/or other facial features captured inthe images, then the flexible illumination process may be done.Otherwise, the controller 360 may select another lighting configuration372, direct the illumination source 330 to illuminate the subjectaccording to the new lighting configuration 372, and direct the camerato capture 342 one or more images with the new lighting configuration372 that are checked for quality according to one or more objectivecriteria. This process may be repeated until a successful authenticationhas been achieved, or for a specified number of attempts until theauthentication attempt is considered failed. In some embodiments, theuser's current pose may be determined by the imaging system andcontroller 360, for example using a gaze tracking algorithm, and theuser's current pose may be used to select an initial lightingconfiguration 372 and, if necessary, one or more subsequent lightingconfigurations 372 for the biometric authentication process.

A similar method may be applied in a gaze tracking process in whichdifferent lighting configurations 372 are selected by the controller 360to obtain better images of the desired features of the user's eyes(e.g., glints) at different poses and in different conditions using oneor more objective criteria.

FIG. 4 is a flowchart of a method for providing flexible illumination inan imaging system, according to some embodiments. As indicated at 400,two or more lighting configurations may be generated and stored to amemory. In some embodiments, a lookup table that maps poses to lightingconfigurations may also be generated and stored. As indicated at 410, aninitial lighting configuration may be selected. As indicated at 420, oneor more images may be captured with the current lighting configurationand analyzed according to one or more objective criterial. At 430, ifthe image quality is determined to be not good enough for the algorithmthat uses the images (e.g., a biometric authentication algorithm)according to the objective criteria, then another lighting configurationmay be selected as indicated at 440, and the method returns to element420 to capture and check additional images. At 430, if the image qualityis determined to be good for the algorithm that uses the images (e.g., abiometric authentication algorithm), then the images may be processed bythe algorithm as indicated at 450. At 460, if more images need to beprocessed (e.g., if the biometric authentication algorithm could notmake an identification based on the images at 450), then the methodreturns to element 420. Otherwise, the method is done.

Biometric Authentication Using Multiple Biometric Aspects

Embodiments of methods and apparatus for biometric authentication aredescribed in which two or more biometric features or aspects arecaptured and analyzed individually or in combination to identify andauthenticate a person. Conventionally, biometric authentication has beenperformed using a single biometric feature. For example, an image of aperson's iris is captured and compared to a baseline image of the user'siris to identify and authenticate the person. In embodiments, an imagingsystem, for example as illustrated in FIGS. 1A through 1D, is used tocapture images of a person's iris, eye, periorbital region, and/or otherregions of the person's face, and two or more features from the capturedimages are analyzed individually or in combination to identify andauthenticate the person (or to detect attempts to spoof the biometricauthentication). Embodiments may improve the performance of biometricauthentication systems, and may help to reduce false positives and falsenegatives by the biometric authentication algorithms, when compared toconventional systems that rely on only one feature for biometricauthentication. Embodiments may be especially advantageous in imagingsystems that have challenging hardware constrains (point of view,distortions, etc.) for individual biometric aspects or features (e.g.,the iris) as additional biometric features (e.g., veins in the eye,portions or features of the periorbital region, or features of otherparts of the face) may be used for biometric authentication if goodimages of one or more of the biometric features cannot be captured at aparticular pose or under current conditions.

The biometric aspects that are used may include one or more of facial,periocular, or eye aspects. For each biometric aspect, one or moredifferent features may be used to describe or characterize the aspect;the different features may, for example, include geometric features,qualitative features, and low-level, intermediate, or high-level 3Drepresentations. The biometric aspects and features may include, but arenot limited to, one or more of the eye surface, eye veins, eyelids,eyebrows, skin features, and nose features, as well as features of theiris such as color(s), pattern(s), and 3D musculature. In someembodiments, feature sizes and geometric relations to other features maybe included as biometric aspects.

FIGS. 5A and 5B illustrate a biometric authentication system thatcombines different biometric aspects, according to some embodiments.FIG. 5A illustrates an example imaging system that combines differentbiometric aspects, according to some embodiments. The imaging system mayinclude, but is not limited to, one or more cameras 540, an illuminationsource 530, and a controller 560. In this example, the eye camera 540 ispointed towards the eye 592, periorbital region 580, and portions of theface 582 to receive reflected light from the illumination source 530.Note, however, in some embodiments, the eye camera 540 may image areflection off a hot mirror as shown in FIG. 1B. Further, in someembodiments, the eye camera 540 may image the user's facial regionincluding the eye 592 through one or more intermediate optical elementsas shown in FIG. 1C. The eye camera(s) 540 may capture 542 individualimages of, or images that include, two or more biometric aspects of theeye 592, periorbital region 580, and portions of the face 582. Thecaptured image(s) may be processed by controller 560 to analyze thequality of two or more of the biometric aspects captured in theimage(s). Depending on the particular application, the controller 560may select a best biometric aspect or feature from the images to be usedfor biometric authentication, or may select two or more of the biometricaspects or features to be used in combination for biometricauthentication.

FIG. 5B is an illustration of the iris 594 and pupil 596 of the eye. Insome embodiments, features of the iris 594 such as color(s), pattern(s),and a 3D reconstruction of muscle patterns in the iris 594 based on twoor more images may be used as biometric aspects or features. An iris 594feature may be used alone, in combination with one or more iris 594features, or in combination with one or more other features of the eye592, periorbital region 580, or face 582 to perform biometricauthorization.

FIG. 6 is a flowchart of a method for performing biometricauthentication using multiple biometric aspects, according to someembodiments. As indicated at 600, one or more images of the user's iris594, eye 592, periorbital region 580, and/or face 582 may be captured byone or more eye cameras. As indicated at 610, the images may beprocessed to extract two or more biometric aspects of the user's iris594, eye 592, periorbital region 580, and/or face 582. As indicated at620, one or more of the biometric aspects may be selected forauthentication. For example, objective criteria may be applied to theextracted biometric aspects to determine whether the biometric aspectsmeet thresholds of quality for the biometric authentication algorithms.One or more of the biometric aspects that meet respective thresholds maythen be selected. As indicated at 630, biometric authentication may thenbe performed using the selected biometric aspect(s).

Biometric Imaging System Using Multiple Cameras

Embodiments of methods and apparatus for biometric authentication aredescribed in which two or more cameras are used to capture images ofbiometric features or aspects for analysis to identify and authenticatea person. Conventionally, biometric authentication has been performedusing a single camera to capture images of biometric features. Forexample, an image of a person's iris is captured by a single eye cameraand compared to a baseline image of the user's iris to identify andauthenticate the person. In embodiments, an imaging system, for exampleas illustrated in FIGS. 1A through 1D, includes at least two camerasthat are used to capture images of a person's iris, eye, periorbitalregion, and/or other regions of the person's face, and one or morefeatures from the captured images are analyzed to identify andauthenticate the person (or to detect attempts to spoof the biometricauthentication).

Embodiments may, for example, be used to capture images of the user'siris using two or more eye cameras for biometric authentication. In someembodiments, instead of or in addition to the iris, two or more camerasmay be used to capture biometric aspects or features of the eye,periorbital region, or user's face including but not limited to the eyesurface, eye veins, eyelid, eye brows, skin, or nose, and use thebiometrics alone or in combination to perform biometric authentication.In some embodiments, feature sizes and geometric relations to otherfeatures may be included as biometric aspects.

Embodiments of biometric systems or algorithms may use images from atleast one of the two or more cameras (two or more per eye, in somesystems) that capture images from different viewpoints of the user'seye, periorbital region, or face to perform biometric authentication. Inconventional biometric systems, typically a single camera is pointeddirectly at the eye region. However, in some compact systems such asHMDs, with an eye camera, the optical path to the target region may bemore complex, with other elements such as lenses or hot mirrors on ornear the optical path, and thus the visibility of target aspects orfeatures may be impaired, and the quality of the captured images may beless than optimal for the biometric authentication algorithms. Adding atleast one additional camera per eye may, for example, allow the imagingsystem to capture images of the eye region from different angles, andallow for switching to a more favorable point of view (pose as locationand orientation), and in some embodiments may allow for two or moreimages captured by two or more cameras to be combined for use inbiometric authentication.

In some embodiments, an algorithm executing on a controller coupled tothe two more cameras may dynamically determine which image(s) capturedby the two or more cameras are to be used for biometric authentication,for example using one or more objective criteria to evaluate the qualityof the captured images. The objective criteria may include one or moreof, but are not limited to, exposure, contrast, shadows, edges,undesirable streaks, occluding objects, sharpness, uniformity ofillumination, absence of undesired reflections, etc. In addition,properties of the region being captured by a camera may be evaluated todetermine quality, for example an overlap of a part of the eye by aneyelid may obscure at least part of a feature in an image captured byone camera while the feature is more visible in an image captured by asecond camera.

In some embodiments, an algorithm executing on a controller coupled tothe two more cameras may combine information from two or more images ofan eye, the periorbital region, or portions of the face captured by atleast two cameras to process aspects and features extracted from thecombined images. The combination of information from two or more imagesmay be performed at different stages of processing. For example, in someembodiments, two or more images may be combined early in processing toenhance the image quality of the resulting combined image from whichaspects or features are extracted and evaluated. As another example, twoor more images may be processed to extract aspects, features or otherinformation in an intermediate stage; the extracted information may thenbe processed in combination to determine a biometric authenticationscore. As yet another example, the information extracted from two ormore images may be processed separately, and then combined in thecomputation of a final similarity/matching score.

FIG. 7 illustrates a biometric authentication system that uses multiplecameras, according to some embodiments. An imaging system may include,but is not limited to, two or more cameras 740, an illumination source730, and a controller 760. In this example, the eye cameras 540 are eachpointed towards the eye 792, periorbital region 780, and/or portions ofthe face 782 to receive reflected light from the illumination source730. Each camera 740 has a different perspective or viewing angle. Alsonote that, while not shown, each camera 740 may center on or capture adifferent feature, aspect, or region of the user's face or eye 792. Insome embodiments, at least one eye camera 740 may image a reflection offa hot mirror as shown in FIG. 1B. Further, in some embodiments, at leastone eye camera 740 may image the user's facial region including the eye792 through one or more intermediate optical elements as shown in FIG.1C. Each eye camera 740 may capture 742 images of, or images thatinclude, one or more biometric aspects of the eye 792, periorbitalregion 780, and portions of the face 782. The images captured by the twoor more cameras 740 may be processed by controller 760 to analyze thequality the image(s). Depending on the particular application, thecontroller 560 may select one or more of the images to be used forbiometric authentication, or may select two or more of the biometricaspects or features from one or more of the images to be used incombination for biometric authentication.

FIG. 8A is a flowchart of a method for biometric authentication usingmultiple cameras, according to some embodiments. As indicated at 800,two or more images of the user's eye, periorbital region, or portions ofthe user's face are captured by two or more cameras. As indicated at802, the captured images are analyzed using one or more objectivecriteria to determine a best image to use for biometric authentication.As indicated at 804, biometric authentication is performed using theselected image.

FIG. 8B is a flowchart of another method for biometric authenticationusing multiple cameras, according to some embodiments. As indicated at820, two or more images of the user's eye, periorbital region, orportions of the user's face are captured by two or more cameras. Asindicated at 822, information from two or more of the images is mergedor combined. As indicated at 824, biometric authentication is performedusing the merged image information.

The merging of information from two or more images may be performed atdifferent stages of processing. For example, in some embodiments, two ormore images may be combined early in processing to enhance the imagequality of the resulting combined image from which aspects or featuresare to be extracted and evaluated. As another example, two or moreimages may be processed to extract aspects, features or otherinformation in an intermediate stage; the extracted information may thenbe processed in combination to determine a biometric authenticationscore. As yet another example, the information extracted from two ormore images may be processed separately, and then combined in thecomputation of a biometric authentication score.

Biometric Imaging Systems Including Additional Optical Elements

Embodiments of methods and apparatus for biometric authentication aredescribed in which one or more additional optical elements are on theoptical path from the illumination system, to the eye or eye region, andthen to the eye camera.

In some embodiments, one or more optical elements such as a lens 120 asshown in FIG. 1C may be on the optical path between the eye 192 and thecamera 140. The optical element may have optical properties; in someembodiments the optical properties may be particular to a user, such asdiopter. In some embodiments, a user may add an extra optical element,such as prescription clip-on lens, to the device's optical system. Theintervening optical element(s) necessarily affect light that passesthrough the element(s) to the camera. In some embodiments, informationabout the optical properties of the intervening optical element(s) maybe obtained and stored, and the controller may adjust images captured bythe camera(s) according to the information to improve image quality foruse in biometric authentication.

In some embodiments, one or more optical elements such as lenses,prisms, diffraction gratings, or waveguides may be located on theoptical path of the eye camera, for example in front of the camera andbetween the camera and the eye/eye region. In some devices, for examplein a HMD with limitations for where eye cameras can be placed, an eyecamera may view the eye or eye region from a non-optimal angle due tothe physical configuration and limitations of the device the imagingsystem is implemented in. An image plane formed at the camera at thenon-optimal angle may affect the quality of the captured images, forexample by reducing pixel density. An optical element such as a lens,prism, diffraction grating, or waveguide on the optical path between theeye/eye region and the eye camera may, for example, be used to “bend”the light rays coming from the eye/eye region, and thus tilt the imageplane, to obtain better pixel density at the eye camera. In other words,the intervening optical element may compensate for perspectivedistortion caused by the camera's position. The intervening opticalelement may thus increase or improve the image space properties of theimaging system.

FIG. 9A illustrates a system that includes at least one additionaloptical element on the light path between the user's eye and the eyecamera, according to some embodiments. An imaging system may include,but is not limited to, one or more cameras 940, an illumination source930, and a controller 960. In this example, the eye camera 940 ispointed towards the eye 992; note, however, that an eye camera 940 mayalso or instead capture images of the periorbital region or portions ofthe face to receive reflected light from the illumination source 930.Note, however, in some embodiments, the eye camera 940 may image areflection off a hot mirror as shown in FIG. 1B. The eye camera 940 mayimage the user's facial region including the eye 992 through one or moreintermediate optical elements 920A and 920B. Element 920A represents alens that is a component of an optical system implemented in the device,and may, but does not necessarily, have optical properties particular toa user. Element 920B represents an optional optical element, such as aclip-on lens, that has been added to an optical system implemented inthe device, and may, but does not necessarily, have optical propertiesparticular to a user. The eye camera(s) 940 may capture 942 individualimages of, or images that include, two or more biometric aspects of theeye 992, periorbital region 980, and portions of the face 982. However,the optical path from the eye region to the eye camera(s) 940 passesthrough the intervening optical element 920A and/or optical element920B.

The intervening optical elements 920A and/or 920B necessarily affectlight that passes through the element(s) to the camera 940. In someembodiments, information about the optical properties of the interveningoptical element(s) (optical element description(s) 976) may be obtainedand stored to memory 970, and the controller 960 may adjust imagescaptured by the camera(s) 940 according to the information to improveimage quality for use in biometric authentication.

The captured image(s) may be further processed by controller 960 toanalyze the quality of one or more of the biometric aspects captured inthe image(s). The image(s) or biometric aspect(s) or features(s)extracted from the image(s) may then be used in a biometricauthentication process.

FIG. 9B illustrates a system that includes a diffractive optical elementon the light path between the user's eye and the eye camera to improvethe viewing angle of the camera, according to some embodiments. Animaging system may include, but is not limited to, one or more cameras940, an illumination source 930, and a controller 960. In this example,the eye camera 940 is pointed towards the eye 992; note, however, thatan eye camera 940 may also or instead capture images of the periorbitalregion or portions of the face to receive reflected light from theillumination source 930. Note, however, in some embodiments, the eyecamera 940 may image a reflection off a hot mirror as shown in FIG. 1B.The eye camera 940 may, but does not necessarily image the user's facialregion including the eye 992 through one or more intermediate opticalelements 920. The eye camera(s) 940 may capture 942 individual imagesof, or images that include, two or more biometric aspects of the eye992, periorbital region 980, and portions of the face 982.

One or more optical elements 924 such as lenses, prisms, diffractiongratings, or waveguides may be located on the optical path of the eyecamera 940, for example in front of the camera 940 and between thecamera 940 and the eye 992. In some devices, for example in a HMD withlimitations for where eye cameras 940 can be placed, an eye camera 940may view the eye 992 or eye region from a non-optimal angle due to thephysical configuration and limitations of the device the imaging systemis implemented in. An image plane formed at the camera 940 at thenon-optimal angle may affect the quality of the captured images, forexample by reducing pixel density. An optical element 924 such as alens, prism, diffraction grating, or waveguide on the optical pathbetween the eye 992 and the eye camera 940 may, for example, be used to“bend” the light rays coming from the eye 992, and thus tilt the imageplane, to obtain better pixel density at the eye camera 940. In otherwords, the intervening optical element 924 may compensate forperspective distortion caused by the camera 940's position. Theintervening optical element 924 may thus increase or improve the imagespace properties of the imaging system.

The captured image(s) may be processed by controller 960 to analyze thequality of one or more of the biometric aspects captured in theimage(s). The image(s) or biometric aspect(s) or features(s) extractedfrom the image(s) may then be used in a biometric authenticationprocess.

FIG. 10 is a flowchart of a method for processing images in a systemthat includes at least one additional optical element on the light pathbetween the user's eye and the eye camera, according to someembodiments. As indicated at 1000, properties of one or more additionaloptical elements on the optical path between the eye camera and the eyeor eye region may be obtained and stored as optical element descriptionsto memory. As indicated at 1010, one or more images of the eye or eyeregion may be captured with the eye camera. As indicated at 1020, thecaptured images may be processed by the controller; the optical elementdescription(s) may be applied to the images to adjust the imageprocessing according to the optical properties of the additional opticalelement(s). At 1030, if processing is done, the method ends. Otherwisethe method returns to element 1010.

FIG. 11 is a flowchart of a method for capturing and processing imagesin a system that includes a diffractive optical element on the lightpath between the user's eye and the eye camera to improve the viewingangle of the camera, according to some embodiments. As indicated at1100, light sources (e.g., LEDs) emit light towards the subject's face.As indicated at 1110, a portion of the light reflected off the subject'sface is diffracted towards the camera by an optical element on theoptical path between the subject's eye and the camera. As indicated at1120, one or more images are captured by the camera. As indicated at1130, the images are processed, for example by a biometricauthentication algorithm on a controller of the device that includes theimaging system. At 1140, if processing is done, the method ends.Otherwise the method returns to element 1100.

Biometric Imaging System Using Multiple Wavelengths

Embodiments of methods and apparatus for biometric authentication andanti-spoofing are described in which two or more different wavelengthsare used in the illumination system. In embodiments, the illuminationsource (e.g. a ring of LEDs) may be configured to emit light at two ormore different wavelengths, either continuously or selectively. Forexample, in some embodiments, a wavelength in the mid-800 nm range maybe used for biometric authentication using the iris, and a wavelength inthe mid-900 mm range may be used for anti-spoofing. Anti-spoofing isrelated to biometric authentication in that “spoofing” refers toattempts to trick a biometric authentication system by, for example,presenting a picture or model of a valid user's eye, eye region, or faceas an attempt to “spoof” the biometric authentication system.

In some embodiments, a method may be implemented in which a firstwavelength is emitted by the illumination source for capturing an imageor images for a first portion of algorithmic processing for biometricauthentication, and a second wavelength is emitted by the illuminationsource for capturing another image or images for a second portion ofalgorithmic processing for biometric authentication. In someembodiments, a camera may sequentially capture two or more images atdifferent wavelengths. As an alternative, in some embodiments, a cameramay be configured to concurrently capture two or more images atdifferent wavelengths.

FIGS. 12A through 12C illustrate a system that includes light sourcesthat emit light at multiple wavelengths to sequentially capture imagesat the multiple wavelengths, according to some embodiments.

FIG. 12A shows an example illumination source 1230 that includesmultiple LEDs 1232. In this example, there are eight LEDs 1232 arrangedin a ring. Note, however, that the number and arrangement of the LEDS1232 in an illumination source 1230 may be different. In addition, insome embodiments other light-emitting elements than LEDs may be used. Insome embodiments, some of the LEDs 1232A, represented by the shadedcircles, may be configured to emit light at a first wavelength in the IRrange, including SWIR and/or NIR, for example at 740, 750, 840, 850,940, or 950 nanometers. The other LEDs 1232B, represented by the whitecircles, may be configured to emit light at a different wavelength inthe IR or NIR range. Note that, in some embodiments, more than twowavelengths may be used. Further, in some embodiments, individuallighting elements may be configured to selectively emit light at two ormore different wavelengths.

FIGS. 12B and 12C illustrate an example imaging system that includeslight sources (e.g., LEDs) that emit light at multiple wavelengths,according to some embodiments. The imaging system may include, but isnot limited to, one or more cameras 1240, an illumination source 1230,and a controller 1260. In this example, the eye camera 1240 is pointedtowards the eye 1292 to receive reflected light from the illuminationsource 1230. However, in some embodiments, the eye camera 1240 mayinstead or also capture images of the periorbital region and portions ofthe face. Note that in some embodiments, the eye camera 1240 may image areflection off a hot mirror as shown in FIG. 1B. Further, in someembodiments, the eye camera 1240 may image the eye 1292 through one ormore intermediate optical elements as shown in FIG. 1C.

In FIG. 12A, the eye camera(s) 1240 may capture 1242A individual imagesof the eye 1292 with LEDS 1232A illuminating the eye at a firstwavelength under control 1244A of the controller 1260. In FIG. 12B, theeye camera(s) 1240 may capture 1242B individual images of the eye 1292with LEDS 1232B illuminating the eye at a second wavelength undercontrol 1244B of the controller 1260.

The captured images may be processed by controller 1260 to analyze thequality of one or more of the biometric aspects captured in the images.Depending on the particular application, the controller 1260 may selecta best biometric aspect or feature from the images to be used forbiometric authentication, or may select two or more biometric aspects orfeatures to be used in combination for biometric authentication.

In some embodiments, the first wavelength may be emitted by theillumination source 1230 for capturing an image or images for a firstportion of algorithmic processing for biometric authentication, and thesecond wavelength may be emitted by the illumination source 1230 forcapturing another image or images for a second portion of algorithmicprocessing for biometric authentication. In some embodiments, the firstwavelength may be used to capture images (e.g., of the iris) for use inan anti-spoofing process, and the second wavelength may be used tocapture images (e.g., of the iris) for use in biometric authentication.

FIGS. 13A and 13B illustrate a system that includes a camera with aphotosensor that concurrently captures multiple images at differentwavelengths, according to some embodiments. As illustrated in FIG. 13A,in some embodiments, as an alternative to sequentially capturing imagesat different wavelengths, a camera sensor 1350 may be provided that isconfigured to concurrently capture two (or more) images at differentwavelengths. In this example, every other pixel is configured to capturelight at a particular wavelength. For example, the white pixels 1352Amay be configured to capture light in the mid-800 nm range, and theshaded pixels may be configured to capture light in the mid-900 range.For example, individual filters may be applied to each pixel 1352, witha first filter applied to pixels 1352A and a second filter applied topixels 1352B.

FIG. 13B illustrates an example imaging system that includes lightsources (e.g., LEDs) that emit light at multiple wavelengths, and inwhich the camera includes a camera sensor 1350 that is configured toconcurrently capture two (or more) images at different wavelengths,according to some embodiments. The imaging system may include, but isnot limited to, one or more cameras 1340, an illumination source 1330,and a controller 1360. In this example, the eye camera 1340 is pointedtowards the eye 1392 to receive reflected light from the illuminationsource 1330. However, in some embodiments, the eye camera 1340 mayinstead or also capture images of the periorbital region and portions ofthe face. Note that in some embodiments, the eye camera 1340 may image areflection off a hot mirror as shown in FIG. 1B. Further, in someembodiments, the eye camera 1340 may image the eye 1392 through one ormore intermediate optical elements as shown in FIG. 1C. The illuminationsource 1330 may be configured to emit light at multiple wavelengths, forexample as illustrated in FIG. 12A. The eye camera(s) 1340 mayconcurrently capture at least two images 1342A and 1342B of the eye 1392at the multiple wavelengths using a sensor 1350 as illustrated in FIG.13A with LEDS 1332A and 1332B concurrently illuminating the eye 1392 atboth wavelengths under control 1344 of the controller 1360.

FIG. 14 is a flowchart of a method for sequentially capturing andprocessing images at multiple wavelengths, according to someembodiments. As indicated at 1400, light sources emit light at a firstwavelength towards the user's eyes. As indicated at 1410, the cameracaptures images at the first wavelength. As indicated at 1420, the lightsources emit light at a second wavelength towards the user's eyes. Asindicated at 1430, the camera captures images at the second wavelength.As indicated at 1440, the images are processed. At 1450, if the methodis not done, then the method returns to element 1410. Otherwise, themethod ends.

FIG. 15 is a flowchart of a method for concurrently capturing andprocessing images at multiple wavelengths, according to someembodiments. As indicated at 1500, light sources emit light at multiplewavelengths towards the user's eyes. As indicated at 1510, the cameraconcurrently captures images for each wavelength, for example using aphotosensor 1350 as illustrated in FIG. 13A. As indicated at 1520, theimages are processed. At 1530, if the method is not done, then themethod returns to element 1510. Otherwise, the method ends.

Improving Eye Pose for Biometric Authentication

Embodiments of methods and apparatus for biometric authentication aredescribed in which a current eye pose is determined and evaluated todetermine if the current pose is satisfactory, and in which the eye posemay be improved by the user manually adjusting the device or theirpose/gaze direction in response to a signal from the controller, and/orin which the imaging system is mechanically adjusted at the direction ofthe controller to improve the current view of the eye.

In embodiments, a method executed on the controller may identify theuser's current eye location and/or orientation (pose), for example bycapturing and evaluating one or more images of the eye(s). Thecontroller may then evaluate how beneficial the current pose is forbiometric authentication. In some embodiments, the controller mayprovide feedback to the user to prompt the user to adjust their pose(e.g., by changing the direction of their gaze) or to manually adjustthe device (e.g., by manually moving the device's position in relationto their eyes). In some embodiments, instead or in addition to promptingthe user to manually adjust their pose or the device, the controller maydirect the imaging system hardware to mechanically adjust the imagingsystem, for example by slightly moving or tilting the camera, or byzooming in or out. Adjusting the pose of the user with respect to theimaging system manually or mechanically may ensure a desired levelbiometric authentication performance, as better images of the eye or eyeregion may be captured. Feedback to the user may be a haptic, audio, orvisual signal, or a combination of two or more haptic, audio, or visualsignals. The automatic adjustment of the imaging system directed by thecontroller may move a component or a combination of components, forexample a module that includes at least the camera. The manual orautomatic adjustments may be a single step in the biometricauthentication process, or alternatively may be performed in a controlloop until certain qualities or objective criteria are achieved in thecaptured images.

FIG. 16 illustrates a system that provides feedback to the user and/orcontrol signals to the imaging system to manually or mechanically adjustthe viewing angle of the camera with respect to the user's eye orperiocular region, according to some embodiments. The imaging system mayinclude, but is not limited to, one or more cameras 1640, anillumination source 1630, and a controller 1660. In this example, theeye camera 1640 is pointed towards the eye 1692 to receive reflectedlight from the illumination source 1630. However, in some embodimentsthe eye camera 1640 may instead or also capture images of theperiorbital region and/or portions of the face. Note, however, in someembodiments, the eye camera 1640 may image a reflection off a hot mirroras shown in FIG. 1B. Further, in some embodiments, the eye camera 1640may image the user's eye 1692 through one or more intermediate opticalelements as shown in FIG. 1C. The eye camera(s) 1640 may capture 1642one or more images of the user's eye 1692. The captured image(s) may beprocessed by controller 1660 to determine a current eye pose and todetermine if the current eye pose is satisfactory for the biometricauthentication process. If the eye pose is not satisfactory, then thecontroller 1660 may provide feedback 1662 to the user to prompt the userto change their eye pose and/or to manually adjust the device. In someembodiments, instead of or in addition to the feedback 1662, thecontroller 1660 may signal 1646 the imaging system to mechanicallyadjust the imaging system, for example by moving or tilting the camera1640.

FIG. 17 is a flowchart of a method for providing feedback to the user tomanually adjust the viewing angle of the camera with respect to theuser's eye or periocular region, according to some embodiments. Themethod may, for example, be performed in a biometric authenticationprocess. As indicated at 1700, the camera captures image(s) of theuser's eye region. As indicated at 1710, the controller determines fromthe image(s) if the alignment of the camera with the desired feature(s)is good. At 1720, if the alignment is not good, the controller mayprompt the user to adjust the gaze and/or to manually adjust the deviceto obtain a better viewing angle, and the method returns to element1700. At 1720, if the alignment is good, then one or more image(s) maybe processed as indicated at 1740. At 1750, if not done processing, thenthe method returns to 1700. Otherwise, the method is done.

FIG. 18 is a flowchart of a method for providing control signals to theimaging system to mechanically adjust the viewing angle of the camerawith respect to the user's eye or periocular region, according to someembodiments. The method may, for example, be performed in a biometricauthentication process. As indicated at 1800, the camera capturesimage(s) of the user's eye region. As indicated at 1810, the controllerdetermines from the image(s) if the alignment of the camera with thedesired feature(s) is good. At 1820, if the alignment is not good, thecontroller may signal the imaging system to mechanically adjust thedevice/camera to obtain a better viewing angle, and the method returnsto element 1800. At 1820, if the alignment is good, then one or moreimage(s) may be processed as indicated at 1840. At 1850, if not doneprocessing, then the method returns to 1800. Otherwise, the method isdone.

Example Systems

FIGS. 19A and 19B are block diagrams illustrating a device that mayinclude components and implement methods as illustrated in FIGS. 1through 18 , according to some embodiments. An example application ofthe methods for improving the performance of imaging systems used inbiometric authentication processes as described herein is in a handhelddevice 3000 such as smartphone, pad, or tablet. FIG. 19A shows a sideview of an example device 3000, and FIG. 19B shows an example top viewof the example device 3000. Device 3000 may include, but is not limitedto, a display screen (not shown), a controller 3060 comprising one ormore processors, memory 3070, pose, motion, and orientation sensors (notshown), and one or more cameras or sensing devices such as visible lightcameras and depth sensors (not shown). A camera 3080 and illuminationsource 3040 as described herein may be attached to or integrated in thedevice 3000, and the device 3000 may be held and positioned by the userso that the camera 3080 can capture image(s) of the user's eye or eyeregion while illuminated by the illumination source 3050. The capturedimages may, for example, be processed by controller 3060 to authenticatethe person, for example via an iris authentication process.

Note that device 3000 as illustrated in FIGS. 19A and 19B is given byway of example, and is not intended to be limiting. In variousembodiments, the shape, size, and other features of a device 3000 maydiffer, and the locations, numbers, types, and other features of thecomponents of a device 3000 may vary.

FIG. 20 illustrates an example head-mounted device (HMD) that mayinclude components and implement methods as illustrated in FIGS. 1through 18 , according to some embodiments. The HMD 4000 may, forexample be a component in an extended reality (XR) system. Note that HMD4000 as illustrated in FIG. 20 is given by way of example, and is notintended to be limiting. In various embodiments, the shape, size, andother features of an HMD 4000 may differ, and the locations, numbers,types, and other features of the components of an HMD 4000 may vary. Insome embodiments, HMD 4000 may include, but is not limited to, a displayand two optical lenses (eyepieces) (not shown), mounted in a wearablehousing or frame. As shown in FIG. 20 , HMD 4000 may be positioned onthe user's head 4090 such that the display and eyepieces are disposed infront of the user's eyes 4092. The user looks through the eyepieces 4020onto the display. HMD 4000 may also include sensors that collectinformation about the user's environment (video, depth information,lighting information, etc.) and about the user (e.g., eye trackingsensors). The sensors may include, but are not limited to one or moreeye cameras 4040 (e.g., infrared (IR) cameras) that capture views of theuser's eyes 4092, one or more scene (visible light) cameras (e.g., RGBvideo cameras) that capture images of the real world environment in afield of view in front of the user (not shown), and one or more ambientlight sensors that capture lighting information for the environment (notshown).

A controller 4060 for the MR system may be implemented in the HMD 4000,or alternatively may be implemented at least in part by an externaldevice (e.g., a computing system) that is communicatively coupled to HMD4000 via a wired or wireless interface. Controller 4060 may include oneor more of various types of processors, image signal processors (ISPs),graphics processing units (GPUs), coder/decoders (codecs), and/or othercomponents for processing and rendering video and/or images. Controller4060 may render frames (each frame including a left and right image)that include virtual content based at least in part on inputs obtainedfrom the sensors, and may provide the frames to the display. FIG. 21further illustrates components of an HMD and MR system, according tosome embodiments.

In some embodiments, an imaging system for the MR system may include,but is not limited to, one or more eye cameras 4040 and an IR lightsource 4030. IR light source 4030 (e.g., IR LEDs) may be positioned inthe HMD 4000 (e.g., around the eyepieces 4020, or elsewhere in the HMD4000) to illuminate the user's eyes 4092 with IR light. At least one eyecamera 4040 (e.g., an IR camera, for example a 400×400 pixel countcamera or a 600×600 pixel count camera, that operates at 850 nm or 940nm, or at some other IR wavelength or combination of wavelengths, andthat captures frames, for example at a rate of 60-120 frames per second(FPS)), is located at each side of the user 4090's face. In variousembodiments, the eye cameras 4040 may be positioned in the HMD 4000 oneach side of the user 4090's face to provide a direct view of the eyes4092, a view of the eyes 4092 through the eyepieces 4020, or a view ofthe eyes 4092 via reflection off hot mirrors or other reflectivecomponents. Note that the location and angle of eye camera 4040 is givenby way of example, and is not intended to be limiting. While FIG. 20shows a single eye camera 4040 located on each side of the user 4090'sface, in some embodiments there may be two or more eye cameras 4040 oneach side of the user 4090's face.

A portion of IR light emitted by light source(s) 4030 reflects off theuser 4090's eyes and is captured by the eye cameras 4040 to image theuser's eyes 4092. Images captured by the eye cameras 4040 may beanalyzed by controller 4060 to detect features (e.g., pupil), position,and movement of the user's eyes 4092, and/or to detect other informationabout the eyes 4092 such as pupil dilation. For example, the point ofgaze on the display may be estimated from the eye tracking; theestimated point of gaze may be used to cause the scene camera(s) of theHMD 4000 to expose images of a scene based on a region of interest (ROI)corresponding to the point of gaze As another example, the estimatedpoint of gaze may enable gaze-based interaction with content shown onthe display. As another example, in some embodiments, brightness of thedisplayed images may be modulated based on the user's pupil dilation asdetermined by the imaging system. The HMD 4000 may implement one or moreof the methods for improving the performance of the imaging systems usedin biometric authentication or gaze tracking processes as illustrated inFIGS. 1 through 18 to capture and process images of the user's eyes4090.

Embodiments of an HMD 4000 as illustrated in FIG. 20 may, for example,be used in augmented or mixed (AR) applications to provide augmented ormixed reality views to the user 4090. HMD 4000 may include one or moresensors, for example located on external surfaces of the HMD 4000, whichcollect information about the user 4090's external environment (video,depth information, lighting information, etc.); the sensors may providethe collected information to controller 4060 of the MR system. Thesensors may include one or more visible light cameras (e.g., RGB videocameras) that capture video of the user's environment that may be usedto provide the user 4090 with a virtual view of their real environment.In some embodiments, video streams of the real environment captured bythe visible light cameras may be processed by the controller 4060 of theHMD 4000 to render augmented or mixed reality frames that includevirtual content overlaid on the view of the real environment, and therendered frames may be provided to the HMD 4000's display system.

FIG. 21 is a block diagram illustrating an example MR system that mayinclude components and implement methods as illustrated in FIGS. 1through 18 , according to some embodiments. In some embodiments, a MRsystem may include an HMD 5000 such as a headset, helmet, goggles, orglasses. HMD 5000 may implement any of various types of displaytechnologies. For example, the HMD 5000 may include a display systemthat displays frames including left and right images on screens ordisplays (not shown) that are viewed by a user through eyepieces (notshown). The display system may, for example, be a DLP (digital lightprocessing), LCD (liquid crystal display), or LCoS (liquid crystal onsilicon) technology display system. To create a three-dimensional (3D)effect in a 3D virtual view, objects at different depths or distances inthe two images may be shifted left or right as a function of thetriangulation of distance, with nearer objects shifted more than moredistant objects. Note that other types of display systems may be used insome embodiments.

In some embodiments, HMD 5000 may include a controller 5060 configuredto implement functionality of the MR system and to generate frames (eachframe including a left and right image) that are provided to the HMD'sdisplays. In some embodiments, HMD 5000 may also include a memory 5062configured to store software (code 5064) of the MR system that isexecutable by the controller 5060, as well as data 5068 that may be usedby the MR system when executing on the controller 5060. In someembodiments, HMD 5000 may also include one or more interfaces (e.g., aBluetooth technology interface, USB interface, etc.) configured tocommunicate with an external device via a wired or wireless connection.In some embodiments, at least a part of the functionality described forthe controller 5060 may be implemented by the external device. Theexternal device may be or may include any type of computing system orcomputing device, such as a desktop computer, notebook or laptopcomputer, pad or tablet device, smartphone, hand-held computing device,game controller, game system, and so on.

In various embodiments, controller 5060 may be a uniprocessor systemincluding one processor, or a multiprocessor system including severalprocessors (e.g., two, four, eight, or another suitable number).Controller 5060 may include central processing units (CPUs) configuredto implement any suitable instruction set architecture, and may beconfigured to execute instructions defined in that instruction setarchitecture. For example, in various embodiments controller 5060 mayinclude general-purpose or embedded processors implementing any of avariety of instruction set architectures (ISAs), such as the x86,PowerPC, SPARC, RISC, or MIPS ISAs, or any other suitable ISA. Inmultiprocessor systems, each of the processors may commonly, but notnecessarily, implement the same ISA. Controller 5060 may employ anymicroarchitecture, including scalar, superscalar, pipelined,superpipelined, out of order, in order, speculative, non-speculative,etc., or combinations thereof. Controller 5060 may include circuitry toimplement microcoding techniques. Controller 5060 may include one ormore processing cores each configured to execute instructions.Controller 5060 may include one or more levels of caches, which mayemploy any size and any configuration (set associative, direct mapped,etc.). In some embodiments, controller 5060 may include at least onegraphics processing unit (GPU), which may include any suitable graphicsprocessing circuitry. Generally, a GPU may be configured to renderobjects to be displayed into a frame buffer (e.g., one that includespixel data for an entire frame). A GPU may include one or more graphicsprocessors that may execute graphics software to perform a part or allof the graphics operation, or hardware acceleration of certain graphicsoperations. In some embodiments, controller 5060 may include one or moreother components for processing and rendering video and/or images, forexample image signal processors (ISPs), coder/decoders (codecs), etc.

Memory 5062 may include any type of memory, such as dynamic randomaccess memory (DRAM), synchronous DRAM (SDRAM), double data rate (DDR,DDR2, DDR3, etc.) SDRAM (including mobile versions of the SDRAMs such asmDDR3, etc., or low power versions of the SDRAMs such as LPDDR2, etc.),RAMBUS DRAM (RDRAM), static RAM (SRAM), etc. In some embodiments, one ormore memory devices may be coupled onto a circuit board to form memorymodules such as single inline memory modules (SIMMs), dual inline memorymodules (DIMMs), etc. Alternatively, the devices may be mounted with anintegrated circuit implementing system in a chip-on-chip configuration,a package-on-package configuration, or a multi-chip moduleconfiguration.

In some embodiments, the HMD 5000 may include one or more sensors thatcollect information about the user's environment (video, depthinformation, lighting information, etc.). The sensors 500 may providethe information to the controller 5060 of the MR system. In someembodiments, the sensors may include, but are not limited to, visiblelight cameras (e.g., video cameras) and ambient light sensors.

HMD 5000 may be positioned on the user's head such that the displays andeyepieces are disposed in front of the user's eyes 5092A and 5092B. IRlight sources 5030A and 5030B (e.g., IR LEDs) may be positioned in theHMD 5000 (e.g., around the eyepieces, or elsewhere in the HMD 5000) toilluminate the user's eyes 5092A and 5092B with IR light. Eye cameras5040A and 5040B (e.g., IR cameras, for example 400×400 pixel countcameras or 600×600 pixel count cameras that operate at 850 nm or 940 nm,or at some other IR wavelength, and that capture frames, for example ata rate of 60-120 frames per second (FPS)), may be located at each sideof the user's face. In various embodiments, the eye cameras 5040 may bepositioned in the HMD 5000 to provide a direct view of the eyes 5092, aview of the eyes 5092 through the eyepieces 5020, or a view of the eyes5092 via reflection off hot mirrors or other reflective components. Notethat the location and angle of eye cameras 5040A and 5040B is given byway of example, and is not intended to be limiting. In some embodiments,there may be a single eye camera 5040 located on each side of the user'sface. In some embodiments there may be two or more eye cameras 5040 oneach side of the user's face. For example, in some embodiments, awide-angle camera 5040 and a narrower-angle camera 5040 may be used oneach side of the user's face. A portion of IR light emitted by lightsources 5030A and 5030B reflects off the user's eyes 5092A and 5092B isreceived at respective eye cameras 5040A and 5040B, and is captured bythe eye cameras 5040A and 5040B to image the user's eyes 5092A and5092B. Eye information captured by the cameras 5040A and 5040B may beprovided to the controller 5060. The controller 5060 may analyze the eyeinformation (e.g., images of the user's eyes 5092A and 5092B) todetermine eye position and movement and/or other features of the eyes5092A and 5092B. In some embodiments, to accurately determine thelocation of the user's eyes 5092A and 5092B with respect to the eyecameras 5040A and 5040B, the controller 5060 may perform a 3Dreconstruction using images captured by the eye cameras 5040A and 5040Bto generate 3D models of the user's eyes 5092A and 5092B. The 3D modelsof the eyes 5092A and 5092B indicate the 3D position of the eyes 5092Aand 5092B with respect to the eye cameras 5040A and 5040, which allowseye tracking algorithms executed by the controller to accurately trackeye movement. The HMD 4000 may implement one or more of the methods forimproving the performance of the imaging systems used in biometricauthentication or gaze tracking processes as illustrated in FIGS. 1through 18 to capture and process images of the user's eyes 4090.

The eye information obtained and analyzed by the controller 5060 may beused by the controller in performing various VR or AR system functions.For example, the point of gaze on the displays may be estimated fromimages captured by the eye cameras 5040A and 5040B; the estimated pointof gaze may be used to cause the scene camera(s) of the HMD 5000 toexpose images of a scene based on a region of interest (ROI)corresponding to the point of gaze. As another example, the estimatedpoint of gaze may enable gaze-based interaction with virtual contentshown on the displays. As another example, in some embodiments,brightness of the displayed images may be modulated based on the user'spupil dilation as determined by the imaging system.

In some embodiments, the HMD 5000 may be configured to render anddisplay frames to provide an augmented or mixed reality (MR) view forthe user based at least in part according to sensor inputs. The MR viewmay include renderings of the user's environment, including renderingsof real objects in the user's environment, based on video captured byone or more video cameras that capture high-quality, high-resolutionvideo of the user's environment for display. The MR view may alsoinclude virtual content (e.g., virtual objects, virtual tags for realobjects, avatars of the user, etc.) generated by MR system andcomposited with the displayed view of the user's real environment.

Embodiments of the HMD 5000 as illustrated in FIG. 21 may also be usedin virtual reality (VR) applications to provide VR views to the user. Inthese embodiments, the controller 5060 of the HMD 5000 may render orobtain virtual reality (VR) frames that include virtual content, and therendered frames may be displayed to provide a virtual reality (asopposed to mixed reality) experience to the user. In these systems,rendering of the VR frames may be affected based on the point of gazedetermined from the imaging system.

Extended Reality

A person can interact with and/or sense a physical environment orphysical world without the aid of an electronic device. A physicalenvironment can include physical features, such as a physical object orsurface. An example of a physical environment is physical forest thatincludes physical plants and animals. A person can directly sense and/orinteract with a physical environment through various means, such ashearing, sight, taste, touch, and smell. In contrast, a person can usean electronic device to interact with and/or sense an extended reality(XR) environment that is wholly or partially simulated. The XRenvironment can include mixed reality (MR) content, augmented reality(AR) content, virtual reality (VR) content, and/or the like. With an XRsystem, some of a person's physical motions, or representations thereof,can be tracked and, in response, characteristics of virtual objectssimulated in the XR environment can be adjusted in a manner thatcomplies with at least one law of physics. For instance, the XR systemcan detect the movement of a user's head and adjust graphical contentand auditory content presented to the user similar to how such views andsounds would change in a physical environment. In another example, theXR system can detect movement of an electronic device that presents theXR environment (e.g., a mobile phone, tablet, laptop, or the like) andadjust graphical content and auditory content presented to the usersimilar to how such views and sounds would change in a physicalenvironment. In some situations, the XR system can adjustcharacteristic(s) of graphical content in response to other inputs, suchas a representation of a physical motion (e.g., a vocal command).

Many different types of electronic systems can enable a user to interactwith and/or sense an XR environment. A non-exclusive list of examplesinclude heads-up displays (HUDs), head mountable systems,projection-based systems, windows or vehicle windshields havingintegrated display capability, displays formed as lenses to be placed onusers' eyes (e.g., contact lenses), headphones/earphones, input systemswith or without haptic feedback (e.g., wearable or handheldcontrollers), speaker arrays, smartphones, tablets, and desktop/laptopcomputers. A head mountable system can have one or more speaker(s) andan opaque display. Other head mountable systems can be configured toaccept an opaque external display (e.g., a smartphone). The headmountable system can include one or more image sensors to captureimages/video of the physical environment and/or one or more microphonesto capture audio of the physical environment. A head mountable systemmay have a transparent or translucent display, rather than an opaquedisplay. The transparent or translucent display can have a mediumthrough which light is directed to a user's eyes. The display mayutilize various display technologies, such as uLEDs, OLEDs, LEDs, liquidcrystal on silicon, laser scanning light source, digital lightprojection, or combinations thereof. An optical waveguide, an opticalreflector, a hologram medium, an optical combiner, combinations thereof,or other similar technologies can be used for the medium. In someimplementations, the transparent or translucent display can beselectively controlled to become opaque. Projection-based systems canutilize retinal projection technology that projects images onto users'retinas. Projection systems can also project virtual objects into thephysical environment (e.g., as a hologram or onto a physical surface).

The methods described herein may be implemented in software, hardware,or a combination thereof, in different embodiments. In addition, theorder of the blocks of the methods may be changed, and various elementsmay be added, reordered, combined, omitted, modified, etc. Variousmodifications and changes may be made as would be obvious to a personskilled in the art having the benefit of this disclosure. The variousembodiments described herein are meant to be illustrative and notlimiting. Many variations, modifications, additions, and improvementsare possible. Accordingly, plural instances may be provided forcomponents described herein as a single instance. Boundaries betweenvarious components, operations and data stores are somewhat arbitrary,and particular operations are illustrated in the context of specificillustrative configurations. Other allocations of functionality areenvisioned and may fall within the scope of claims that follow. Finally,structures and functionality presented as discrete components in theexample configurations may be implemented as a combined structure orcomponent. These and other variations, modifications, additions, andimprovements may fall within the scope of embodiments as defined in theclaims that follow.

What is claimed is:
 1. A system, comprising: a camera configured tocapture images of an object; a controller comprising one or moreprocessors; and an illumination source comprising a plurality oflight-emitting elements, wherein individual ones of the light-emittingelements are configured to be controlled independently of the otherlight-emitting elements to emit light towards the object to be imaged bythe camera; and wherein the controller is configured to: select one of aplurality of different lighting configurations for the illuminationsource; direct the illumination source to emit light according to theselected lighting configuration; and process one or more images of theobject captured by the camera as illuminated by the illumination sourceaccording to the selected lighting configuration.
 2. The system asrecited in claim 1, further comprising a memory that stores theplurality of different lighting configurations for the illuminationsource.
 3. The system as recited in claim 2, wherein the object is aneye or periorbital region of a user of the system, wherein the memoryfurther comprises a lookup table that maps different poses of the userto respective lighting configurations, and wherein, to select one of aplurality of different lighting configurations for the illuminationsource, the controller is configured to: determine a current pose of theuser; and select the lighting configuration from the lookup table basedon the current pose.
 4. The system as recited in claim 3, wherein a poseis a 3D geometrical relationship between the camera and the user'scurrent eye position and gaze direction.
 5. The system as recited inclaim 1, wherein a lighting configuration specifies one or more aspectsof lighting including, but not limited to, light-emitting elements orgroup of light-emitting elements to enable or disable, intensity ofindividual ones or groups of the light-emitting elements, wavelength ofindividual ones or groups of the light-emitting elements, shapes andsizes of the of individual ones or groups of the light-emittingelements, angular profile of individual ones or groups of thelight-emitting elements, direction of individual ones or groups of thelight-emitting elements, and sequences of individual ones or groups ofthe light-emitting elements.
 6. The system as recited in claim 1,wherein, to process one or more images of the object captured by thecamera as illuminated by the illumination source according to theselected lighting configuration, the controller is configured to: checkthe one or more images for quality according to one or more objectivecriteria; and upon determining that the quality of the images is below athreshold for an algorithm to be applied to the images: select adifferent one of the plurality of different lighting configurations forthe illumination source; direct the illumination source to emit lightaccording to the different lighting configuration; and process one ormore images of the object captured by the camera as illuminated by theillumination source according to the different lighting configuration.7. The system as recited in claim 6, wherein the controller is furtherconfigured to, upon determining that the quality of the images is at orabove the threshold for the algorithm to be applied to the images, applythe algorithm to the images.
 8. The system as recited in claim 6,wherein the algorithm is a biometric authentication algorithm.
 9. Thesystem as recited in claim 6, wherein the algorithm is a gaze trackingalgorithm.
 10. The system as recited in claim 6, wherein the objectivecriteria include one or more of exposure, contrast, shadows, edges,undesirable streaks, occluding objects, sharpness, uniformity ofillumination, and absence of undesired reflections.
 11. The system asrecited in claim 1, wherein the light-emitting elements arelight-emitting diodes (LEDs).
 12. The system as recited in claim 1,wherein the light-emitting elements are infrared (IR) light sources, andwherein the camera is an infrared camera.
 13. The system as recited inclaim 1, wherein the system is an imaging system, and wherein the objectis an eye, periorbital region, or portion of a face of a user of thesystem.
 14. The system as recited in claim 13, wherein the imagingsystem is a component of a head-mounted device (HMD), a handheld device,or a wall-mounted device.
 15. A method, comprising: selecting, by acontroller comprising one or more processors, one of a plurality ofdifferent lighting configurations for an illumination source, whereinthe illumination source comprises a plurality of light-emitting elementsthat are configured to be controlled independently to emit light towardsand object to be imaged by a camera; directing, by the controller, theillumination source to emit light towards the object according to theselected lighting configuration; capturing, by the camera, one or moreimages of the objects as illuminated by the illumination sourceaccording to the selected lighting configuration; and processing, by thecontroller, the one or more images of the object captured by the cameraas illuminated by the illumination source according to the selectedlighting configuration.
 16. The method as recited in claim 15, whereinthe object is an eye or periorbital region of a user of the system, andwherein, selecting one of a plurality of different lightingconfigurations for the illumination source comprises: determining acurrent pose of the user; and selecting the lighting configuration basedon the current pose.
 17. The method as recited in claim 16, wherein apose is a 3D geometrical relationship between the camera and the user'scurrent eye position and gaze direction.
 18. The method as recited inclaim 15, wherein a lighting configuration specifies one or more aspectsof lighting including, but not limited to, light-emitting elements orgroup of light-emitting elements to enable or disable, intensity ofindividual ones or groups of the light-emitting elements, wavelength ofindividual ones or groups of the light-emitting elements, shapes andsizes of the of individual ones or groups of the light-emittingelements, angular profile of individual ones or groups of thelight-emitting elements, direction of individual ones or groups of thelight-emitting elements, and sequences of individual ones or groups ofthe light-emitting elements.
 19. The method as recited in claim 15,wherein processing the one or more images of the object captured by thecamera as illuminated by the illumination source according to theselected lighting configuration comprises: checking the one or moreimages for quality according to one or more objective criteria; and upondetermining that the quality of the images is below a threshold for analgorithm to be applied to the images: selecting a different one of theplurality of different lighting configurations for the illuminationsource; directing the illumination source to emit light according to thedifferent lighting configuration; and processing one or more images ofthe object captured by the camera as illuminated by the illuminationsource according to the different lighting configuration.
 20. The methodas recited in claim 19, further comprising, upon determining that thequality of the images is at or above the threshold for the algorithm tobe applied to the images, applying the algorithm to the images.